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\BS TRACT 

This study evaluated the effects of an experimental 
critical thinking program* Students at the sixth through ninth grade 
levels vere randomly assigned to experimental and control groups. 
Experimental subjects received 36 tvo-hour veeks of instruction 
consisting of five independent units of programed booklets, games, 
and activities. Objective-referenced pretests and posttests were 
given to all subjects for each unit and a standardized critical 
thinking and dispositional inventory was administered at the end* 
Significant differences between treatments were found at all grade 
levels for units administered early in the study. However, results 
vere mixed on later measures suggesting a debilitating motivational 
trend. (Author) 
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Abstract 

The present srudy evaluafed the effects of an experimental critical- 
thinking program. Students at the sixth through ninth grade v/ere randomly 
assigned to exper irnenta ! and control groups. Experimental Ss_ received 56 
Two-hour weeks of instruction consisting of five independent units of 
programrred bookieTs, games, and activities. Objective-referenced pre- and 
posttests v\er'e given to all Ss^ for each unit and a standardized critical 
tninking ana dispositional inven+or/ was administered at the end. Significant 
differences between treatments were found at all grade levels for units 
administered eariy in tne study, however, results were mixed on later 
measures suggesfing a debilitating motivational trend. 
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The Effects of Instruction 
on the Critical Thinking Abilities 
of Middle-School-Age Children 

John W. Thomas 
Research for Better Schools, Inc. 

Introduction 

A number of studies have focused on the improvement of children's 
critical thinking abilities. Training studies have taken a variety of forms, 
including programs in inquiry, critical reading, formal logic, problem solving 
and reflective thinking. Curriculum development activities in critical think- 
ing have taken place within all of the major subject matter areas. In his 
review of the processes of thinking, Russell (1965) summarized the diverse 
interpretations given to critical thinking as follows: 

It has cecn raie si/nonynous vith the ability to abstract and 
organize infor^^-ation^ to di^aij inferenceSj to search for rele- 
vant materials^ to evaluate data^ to compare sources^ to em- 
ploy a fovrri-MisGOuri attitude^ to distinguish fact from 
opinion^ to detect propoganda^ and to apply the rules of 
logical reasoning, fp. 24) 

Typically, the investigator interested in fostering critical thinking 
behaviors will define the objectives for an instructional program in terms 
of general abilities or aspects of critical thinking. These aspects may 
refer to activities such as making inferences, testing hypotheses and 
recognizing assumptions or they may refer to types of thinking such as 
conditional and deductive reasoning. For example, Ennis (1962) lists 
twelve aspects of critical thinking, Kolesnik (1962) defines eight con- 
cepts of critical thinking, Raths, Wasserman and Uasserman's (1969) 
curriculum materials focuses on twelve general abilities. The assumption 
is often made that students who receive sufficient practice at identifying 
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problems, drawing concV ions or evaluating syllogisms will be able to 
apply what they have learned to a wide variety of problems and issues. 

Berlak (1965) criticized the "general aspects" view with respect to 
instruction in critical thinking. Berlak questions the value of teaching 
a set of generalized strategies. According to Berlak, the value 
of intellectual skills resides in their applicability to specific problems 
and issues of interest. Berlak recommends tnat investigators interested 
in teaching critical thinking, turn their attention to the thought pro- 
cesses and operations involved in specific areas of human experience. 
Aspects of critical thinking selected as behavioral objeccives should 
"have demonstrated value to persons who have dealt successfully with 
some problem or issue." (p. 3) 

The intent of the present investigation was to evaluate the effec- 
tiveness of a set of instructional materials designed to teach selected 
critical thinking skills to middle-school-age children. Lessons, garr.as 
and activities were developfjd to form five independent courses or units. 
The units introduce standards for making judgments — specif ic principles, 
conventions and criteria for assessing the reliability, relevancy, 
v/arranty or sufficiency of information — within the context of life-like 
problem solving ventures. Each course is constructed around a distinct 
societal decision-making role and problem-solving paradigm: 
1. "Advertising" — the role of the consumer advocate in evaluating per- 
suasive techniques and empirical product claims; 2. "Judging" — thr role 
of the courtroom lawyer in evaluating evidence and testimony; 
3. "Reporting" the role of the investigative reporter in interpreting 
reports and opinions; 4. "Conflict" — the role of the mediator in re- 



solving interpersonal quarrels; .nd 5. "Causation" — the role of the social 
scientist in testing hypotheses anc designing research studies. 

Subjects 

Six schools from the suburban Philadelphia area participated in the 
study. The socio-economic composition of these suburban communities 
ranged from lower-middle to upper-middle class. The building principal 
within each school suggested teachers vno might be willing to take part 
in the study. For the most part, teachers who expressed a willingness 
to cooperate were randomly assigned to either the experimental or control 
condition. Then, students within each grade were randomly assigned to one 
of the two conditions. At the onset of the study, the groups were fairly 
equal in number within each grade. The total population consisted of 
366 sixth-graders, 229 seventh-graders, 195 ef ghth-graders and 260 ninth- 
graders. By the end of the study, multiple absences and the withdrawal 
of one school reduced the total sample from 1,050 to 818 students. 

Materials 

The experimental material was the above-mentioned prototype curriculum 
package entitled MaP.iyig Judar:ents. Makiy^g Judgments is comprised of five 
independent courses each consisting of five to six programmed lesson books 
and workbooks, one to two group board games and several teacher-led group 
activities. The units v/ere constructed to provide a balance of individual 
small-group and large-qroup activities and the sequence of lessons within 
each unit was constructed to proceed from linearly programmed exercises to 
paper and pencil simulations or branched problem-solving tasks. 
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The criterion measures that were of principal interest in the study 
were the objective-referenced posttests that accompanied each course. Each 
of the five posttests consisted of approximately thirty items, usually two 
items for each objective. Some items were constructed in order to assess 
the mastery of concepts and principles while others were written in order 
to measure the extent to which students can transfer what they've learned 
to novel content areas. Transfer items were all open-ended reasoning 
items and did not include any of the course-related vocabulary present in 
many of the mastery items. Each course also included a pretest. Pretests 
were constructed by randomly selecting five mastery and five transfer 
items from the original pool of items. 

Additional criterion measures included Level X of the Cornell Critical 
Thinking Test (Ennis and Millman, 1971) and a dispositional inventory 
made up of A Locus of Control Scale for Children (Nowicki and Strickland, 
1972), Intolerance o^ Ambiguity Scale (Martin and Westie, 1959) and 
Attitude Toward School (Instructional Objectives Exchange, 1972). 

All tests were administered with standard instructions and were scored 
by independent raters using standard scoring instructions. Protocols 
were coded to prevent the raters from knowing a student's name, school 
or experimental condition. 

Formative evaluation questionnaires, interviews and observation scales 
v/ere also used in the study to collect data concerning length and diffi- 
culty of the lesson, ease of administration of the materials, teacher and 
student attitude toward the material and teacher attitude toward the 
evaluation project. 
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P^'ocedure 

An cooDeratinq teachers attended a brief orientation session in late 
September 1973. At that time, teachers v^/ere given an administration schedule 
and a Teacher's Guide for the first of the five units. From October to June, 
experimental teachers administered one unit after another according to a 
suggested schedule of two to three, one-hour sessions per week. Pretests 
were administered before and posttests v/ere administered after each course 
for both the experimental and control groups. Control classes received no 
special treatment and were tested on the same day as the experimental 
class or classes within their school. Throughout the seven and one-half 
month study, a variety of formative evaluation indices were completed by 
teachers, students and observers. The Cornell Critical Thinking Test and 
the dispositional inventory were administered within a week of the post- 
test that accompanied the last of the five units. 

Students who v/ere absent from a session or two of any course 
were given makeup work to do at home or in a free period. Students who 
missed three or more lessons in c" course were excluded from the analysis of 



that unit. The group sizes for the analyses are given in Table 1. 

Table 1 

SAMPLE SIZES FOR STATISTICAL ANALYSES 









TESTS 
















Critical 








Unit A 


Unit B 


Unit C 


Unit D 


Unit E 


Thinking 


Dispos. 


Grade 


Group 


Posttest 


Posttest 


Posttest 


Posttest 


Posttest 


Test 


Inventory 




E 


198 


202 


203 


205 


205 


204 


205 


6 


C 


146 


149 


156 


151 


151 


147 


151 




E 


84 


94 


83 


77 


83 


86 


87 


7 


C 


78 


106 


89 


89 


83 


84 


88 




E 


71 


63 


49 


55 


55 


^,9 


52 


8 


C 


74 


74 


57 


54 


51 


49 


50 




E 


91 


94 


95 


97 


90 


86 


86 


9 


C 


92 


95 


85 


86 


90 


84 


79 
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The Analysis 

A number of research questions were formulated including the 
following: 1) Are the treatment groups within each grade comparable? 
2) What is happening that we need to consider for further modification 
of the materials? 3) How does sex, prior knowledge and intelligence 
affect the use of the materials? 4) Do the materials teach the content 
specified in the behavioral objectives (mastery)? 5) Do the materials 
teach the skills in a way that allows the student to generalize what he 
has learned to other situations (transfer)? 6) Do the students exhibit 
any differences in their disposition to use the skills taught? 7) What 
are the side effects which accompany the use of these materials? 

Cochran's homogeneity of variance analysis v/as conducted on student 
IQ scores obtained from school records and on each of the five pretests 
for all grades and sites. 

Responses to the dispositional inventory and che Cornell Critical 
Thinking Test were hand scored and subjected to analysis of variance 
(grade by treatment). The unit Dosttests were scored, then transferred 
to optical scanning sheets for a two-v/ay analysis of variance conducted 
by computer. 

Data from the formative evaluation instruments were summarized and 
are available elsewhere (Rose, 1974). 

Results 

All comparison groups were found to be homogeneous with respect to 
both IQ scores and pretest scores. 
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The experimental group outperfonned the control group in 16 of the 
20 comparisons made on the subtests of the Cornell Critical Thinking Test. 
In only one instance was a significant difference found between the 
grC'Ups' scores. Ninth grade experimental students scored significantly 
higher than ninth grade controls on the identifications of assumptions 
subtest. 

Insert Table 2 here. 

An analysis of variance performed on the subtes^.s of the dispositional 
inventory showed a significant difference favoring experimental over con- 
trol subjects on the Locus of Control measure in grade 6, on the Intoler- 
ance of Ambiguity measure in grade 7 and on the Attitude Toward School 
measure, also in grade 7, Differences favoring the control group over the 
experimental group were significant on the Attitude Toward School measure 
in grade 6 and on the Locus of Control measure in grade 9, 

Insert Table 3 here. 

Table 4 summarizes the results of the analysis of variance conducted 
on mastery items, transfer items and on total posttes^- scores for each 
course. 

Insert Table 4 here. 

Figure 1 displays the total posttest mean differences between experimen- 
tal and control subjects by grade for all five courses. For the first 
course tested, Advertising, all comparisons are statistically significant 
and favor the experimental students in grades six, eight and nire. 
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TABLE 4 



SUMMARY OF F-RAT^OS FOR 
ANALYSIS OF VARIANCE OF POSTTEST SCORES 
BY GRADE AND COURSE 







GRADE 


COURSE 


SUBTEST 


6 


7 


8 


9 


Advertising 


Mastery (28^) 
Transfer (23) 
Total Posttest 


30.32** 
17.50** 
29.34** 


5.41* 
1 

3.14 


17.48** 
14.00** 
18.55** 


45.86** 
52.06** 
58.25** 


Judging 


Mastery (29) 
Transfer (24) 
Total Posttest 


43.74** 
16.76** 
36.31** 


43.75** 
11.37** 
32.70** 


20.20** 
2.40 
11.14** 


51.17** 
43.30** 
52.51** 


Reporting 


Mastery (48) 
Transfer (24) 
Total Posttest 


22.19** 
4.53* 
20.08** 


1.10 
1 

1.15 


3.44 

5.60* 

4.41* 


1.46 
2.05 
1.70 


Conflict 


Mastery (70) 
Transfer (7) 
Total Posttest 


4.79* 
1 

4.19* 


2.71 

9.43* 

3.68 


4.35* 

3.41 

4.55* 


66.49** 
31.51** 
67.19** 


Causation 


Mastery (31) 
Transfer (20) 
Total Posttest 


2.68 
1.70 
2.55 


1.07 
1.95 
1 


4.18* 

1 

2.80 


15.33** 
12.86** 
15.89** 



a Number of items in subtest. 
* P .05. 
** p .01- 
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Differences with respect to mastery items only were significant in grade 
seven. The results of the second course tested, Judging, showed signifi- 
cant differences favoring the treated students in all comparisons except 
for transfer items in grade eight. For Reporting, the third course tested, 
differences for both subtests and total tests were found to be significant 
in the sixth grade sample. In addition, the results for transfer items 
and total posttest items significantly favored the experimental group in 
grade eight. 

Results for the Conflict unit shov/ed significant differences favoring 
the experimental students on both subtests and total test in grade nine, 
on mastery items and total test in grades six and eight and on transfer 
items in grade seven. With respect to the final unit tested. Causation, 
again all comparisons significantly favored experimental students in grade 
nine, however, only one other comparison was significant. In grade eight, 
experimental students significantly outperformed controls on mastery items. 



Insert Figure 1 here. 

Discussion 

The data lend some support to the premise that middle-school -age child- 
ren can benefit from instructional materials designed to teach critical 
thinking skills in the context of problem solving and decision making ven- 
tures. 

The results of the Cornell Critical Thinking Test and the subtests of 
the dispositional inventory can only be described as equivocal in the first 
instance and uninterpretable in the second. Insofar as these tests were 
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AOVERTtSiNG 



Making Judgments Group 



Control Subjects 





• M J group fd vorrd at p < .05 
•• MJ group f«ivorcd at p<.01 



FIGURE 1 . ' 

ANALYSIS OF VARIANCE AS RELATED TO POSTTEST MEANS FOR 
TOTAL POSTTEST SCORES BY GRADE LEVEL FOR EACH COURSE 



neither designed for this age group nor matched to the objectives of the 
instructional material, the lack of significant outcome was not regarded 
as a qualification of the materials' effectiveness. 

The criterion measures of p^^incipal interest in this study were the. 
mastery and transfer subtests of the objective-referenced posttests. Where- 
as the results of the first two courses tested strongly supported the ef- 
fectiveness of the materials for accomplishing the objectives and fostering 
transfer of training, results from the final three units were less positive 
than expected. 

Results from attitude questionnaires completed periodically by the 
teachers showed an increasing adverse reaction to the requirements and time 
involved in the pilot test that was unrelated. to teachers* consistent 
positive evaluation of the materials. Printing and delivery delays increased 
the time span of the pilot test from four and one-half months to seven and 
one-half months. The decline in teacher morale precipitated by these 
delays may explain, in part, the decline in the effectiveness of units 
administered late in the study. In addition, the difficulty level of the 
last three units. Causation and Conflict in particular, was judged by 
teachers and others to be somewhat higher than the initial units. The fact 
that only ninth grade students outperformed controls on both subtests in 
the last two units lends support to this observation. 

In retrospect, this study would have been more meaningful if all 
courses had Deen tested at the same time with independent samples. Such a 
field test of the package is currently being conducted in Los Angeles, Reno, 
Nevada, Philadelphia and West Germany. The five courses are being administered 
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as mini-courses with an equal amount of individual and teacher-led activities. 
To date, student and teacher attitudes toward the materials and the test 
conditions have been highly positive. 
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